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Background / Context: 


Over the past two decades, the number and types of after-school programs (ASPs) have 
increased substantially as a result of increased federal and private spending and because ASPs 
are perceived to provide wide-ranging and far-reaching benefits to students, families, schools and 
the public (Mahoney et al., 2009). Research on effects of ASPs, however, has produced a body 
of ambiguous evidence. Several reviews and two meta-analyses have been conducted examining 
the outcomes of ASPs (see Apsler, 2009; Durlak et ah, 2010; Fashola, 1998; Hollister, 2003; 
Lauer et ah, 2006; Roth et ah, 2010; Scott-Little, Hamann, & Jurs, 2002; Zief et ah, 2006), 
yielding mixed and inconclusive results for the effects of ASPs on attendance or externalizing 
behavior outcomes. Prior reviews, however, are limited by the quality and methods employed to 
conduct the reviews (e.g., did not use systematic methods, employed a narrative approach, did 
not assess quality of included studies). In addition, most reviews are outdated; the two most 
recently published reviews concluded the search for studies in 2007 (Durlak et ah, 2010; Roth et 
ah, 2010). As the number of studies of ASPs conducted since 2007 has increased, and synthesis 
methods have advanced, it seems timely for an updated systematic review and meta-analysis of 
ASPs to examine the extent to which the outcome research has advanced and examine the effects 
of ASPs using contemporary studies and techniques. 

Purpose / Objective / Research Question / Focus of Study: 

The purpose of this systematic review and meta-analysis is to synthesize the 
available evidence on the effects of after- school programs with at-risk primary and 
secondary students on school attendance and externalizing behavior outcomes. The specific 
questions guiding this review were: 1) What are the effects of ASPs on school attendance with 
at-risk students who attend an ASP compared to at-risk students who do not attend an ASP? 
2) What are the effects of ASPs on externalizing behavioral outcomes with at-risk students 
who attend an ASP compared to at-risk students who do not attend an ASP? 3) Are there 
study, participant or program characteristics that moderate effects of ASPs? 

Setting: 

Studies included in this review were conducted in any setting that housed an ASP (e.g., 
school, community organization, church). Due to significant differences in educational systems 
around the world, this review was limited to studies conducted in the United States, Canada, 
United Kingdom, Ireland, and Australia. 


Population / Participants / Subjects: 

Published and unpublished studies conducted between 1980 and May, 2014 were eligible 
for inclusion if they examined the effects of an ASP with primary or secondary school students 
who were considered “at-risk” defined as: (1) performing below grade level or having low scores 
on academic achievement tests; (2) attending a low-performing or Title I school; (3) having 
characteristics associated with risk for lower academic achievement, such as low socioeconomic 
status, racial- or ethnic-minority background, single-parent family, limited English proficiency, 
or a victim of abuse or neglect; or (4) engaging in high-risk behavior, such as truancy, running 
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away, substance use, or delinquency (adapted from Lauer et al., 2006). To be included in the 
review, studies must have employed an experimental or quasi-experimental design examining 
effects of an ASP against a comparison group (wait-list or no intervention, treatment as usual, 
straw-man, or alternative interventions) and reported baseline measures of outcome variables or 
covariate adjusted posttest means. Studies must have measured either school attendance or 
externalizing behaviors (broadly defined as any acting out or problematic behavior, including but 
not limited to disruptive behavior, substance use, or delinquency) to be included. 

Intervention / Program / Practice: 

Interventions included in this review were ASPs defined as an organized program 
supervised by adults that occurred during after-school hours within the regular school year. 
Interventions that operated solely during the summer, occurred during school hours, or were 
solely mentoring or tutoring services were excluded from this review. Mentoring and tutoring 
programs, while often occurring after school, are not generally classified as an after-school 
program and have been synthesized as separate types of interventions (see DuBois et al., 2011; 
Ritter et al., 2009; Tolan et al., 2013). 

Research Design: 

Systematic review methodology, following the Campbell Collaboration procedures and 
guidelines (Campbell Collaboration, 2014), was used for all aspects of the search, retrieval, 
selection, and coding of published and unpublished studies meeting study inclusion criteria. 
Meta-analysis was used to quantitatively synthesize results across studies. 

Data Collection and Analysis: 

Information Sources. Several sources were used to identify eligible published and 
unpublished studies between 1980 and May, 2014. Eight electronic databases were searched. 
Keyword searches within each electronic database included variations of “after-school program” 
and (evaluation OR treatment OR intervention OR outcome). The full search strategy for each 
electronic database is available from the authors. Potential reports were also sought by searching 
several research registers and internet sites; hand-searching reference lists of prior reviews and 
articles identified during the search; and contacting experts via email for potentially relevant 
published and unpublished reports. 

Study Selection and Data Extraction. Titles and abstracts of the studies found through 
the search procedures were screened for relevance by one author. Documents that were not 
obviously ineligible or irrelevant based on the abstract review were retrieved in full text and 
screened by one author using a screening instrument. Following initial screening, potentially 
eligible studies were further reviewed by two authors to detennine final inclusion. Any 
discrepancies between authors were discussed and resolved through consensus and, when 
needed, a third author reviewed the study. 

Studies that met inclusion criteria were coded using a coding instrument comprised of 
five sections: 1) source descriptors and study context; 2) sample descriptors; 3) intervention 
descriptors; 4) research methods and quality descriptors; and 5) effect size data. The data 
extraction instrument, available from the authors, was pilot tested by two authors and 
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adjustments to the coding form were made. Two authors then independently coded all data 
related to moderator variables (i.e., study design, grade level, contact frequency, control 
treatment, program type, and program focus), risk of bias (Cochrane Collaboration’s Risk of Bias 
Tool; Higgins et ah, 201 1) and effect size calculations. Discrepancies between the two coders 
were discussed and resolved through consensus. Descriptive data related to study, sample and 
intervention characteristics were coded by one author, with 20% of the studies coded by a second 
author. Inter-rater agreement on descriptive items was 92.3%. If data were missing from a study, 
every effort was made to contact the study author to request the missing data; we received 
additional data from four authors. 

Statistical Procedures. We calculated the standardized-mean difference, correcting for 
small-sample bias using Hedges g for each outcome included in the review. To control for pre- 
test difference between the intervention and control conditions, we subtracted the pre-test effect 
size from the post-test effect size (Lipsey & Wilson, 2001). The variance was calculated for each 
effect size, adjusting for the number of effect sizes in the study (Hedges et ah, 2010). Robust 
variance estimation (RVE), was used to synthesize the effect sizes. Unlike traditional meta- 
analysis, RVE allows for the inclusion and synthesis of all estimated effect sizes simultaneously 
(Hedges et ah, 2010; Tanner-Smith & Tipton, 2013). RVE models each of the effect sizes, 
eliminating the need to average or select only one effect size per study. The result of the analysis 
is random-effects weighted average, similar to traditional syntheses, but including all available 
information. Of note, we chose to conduct separate meta-analyses for the attendance and 
behavioral outcomes, given their divergent latent nature. Finally, we estimated the heterogeneity 
and attempted to model it. Higgins and Thompson (2002) suggested the calculation of 7 , which 
quantifies the amount of heterogeneity beyond sample differences. Given sufficient 
heterogeneity, we conducted moderator analyses; we limited the number of moderators to 
decrease the probability of spurious results (Authors, in press). In total, we used six a priori 
detennined variables: age (i.e., elementary, middle, or mixed), amount of program contact (i.e., 
weekly, 3-4 per week, or daily), control group type (i.e., wait list, treatment as usual, straw-man, 
or alternative intervention), study design (i.e., random or non-random), program type (i.e., 
National or other), and program focus (i.e., academic, non-academic, or mixed). We used the R 
package robumeta (Fisher & Tipton, 2014) to conduct all analyses. 

Findings / Results: 

Twenty-four studies reported in 31 reports were included in the review (See Figure 1 for 
study search and selection Flow Chart, Table 1 for summary of included studies, Table 2 for 
study and sample characteristics of included studies and Table 3 for intervention characteristics 
of Included Studies). 

(Please insert Tables 1, 2, & 3 here) 

(Please insert Figure 1 here) 

Attendance. A total of 16 studies including 16 effect sizes were synthesized to capture 
the effects of the interventions on students’ attendance. The results of the synthesis indicated a 
very small, non-statistically significant treatment effect (g = 0.04, 95% C.I. = -0.02, 0.10; see 
Figure 2). The homogeneity analysis indicated a moderate degree of heterogeneity (x 2 = .002, / 2 = 
66.67%). Only five moderator analyses were conducted because the “program focus” variable 
did not include sufficient variability (i.e., all but 1 study used a mixed approach). As presented in 
Table 4, the results of the moderator analyses did not reveal significant differences ip > .05). 
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(Please insert Figure 2 here) 

(Please insert Table 4 here) 

Externalizing behaviors. Sixteen studies, including 49 effect sizes, were synthesized to 
capture effects of interventions on externalizing behavior (mean n of effect sizes = 2.58, Min = 1, 
Max = 10). Most of the effect sizes measured disruptive behavior or delinquency ( n = 39, 79.6%) 
and the rest measured substance use (n = 10, 20.4%). We chose to pool all measures of 
externalizing behaviors rather than separate drug or alcohol usage from other externalizing 
behaviors to allow for greater statistical power and because moderator analysis indicated no 
significant differences in effects of interventions between substance use and other externalizing 
behavior outcomes ( t = 0.84,/? = 0.47). The results of the meta-analysis indicated a small, non- 
significant effect (g = 0.11, 95% C.I. = -0.05, 0.28; See Figure 3). The homogeneity analysis 
indicated a high degree of heterogeneity (x 2 = .03, / 2 = 79.74%). As such, we conducted 
moderator analyses using all six variables. Results of the moderator analyses did not reveal 
significant differences (p > .05; see Table 5). 

(Please insert Figure 3 here) 

Please insert Table 5 here) 


Conclusions: 

ASPs receive overwhelming positive support and significant resources; however, the 
overall lack of rigorous studies assessing effects of ASPs and the lack of significant effects of 
ASPs on attendance and externalizing behaviors found in this review, along with discrepant 
findings of prior reviews, provide some impetus for us to reconsider the purpose of ASPs and the 
way ASPs are designed and implemented. For school attendance, the evidence from this review 
converges with prior quantitative and narrative reviews. Simply, ASPs have not demonstrated 
significant effects on school attendance (Durlak et ah, 2010; Zief et ah, 2006). Similar to 
findings related to effects on attendance, the present review’s findings point to non-significant 
effects of ASPs on externalizing behavior. Although the present results support findings of Zief 
et al.’s (2006) review, the conclusions offered by other prior reviews on behavioral outcomes 
have been more positive (Durlak et ah, 2010; Scott-Little et ah, 2002). The contrast between our 
findings and the more positive findings from prior reviews likely stems from several factors 
(e.g., methodological and statistical procedures and inclusion criteria). 

The evidence on ASPs to date is fraught with methodological shortcomings, limiting 
conclusions that can be drawn about the effects of ASPs. ASPs seem to be expected to affect 
numerous outcomes, but attempt to do so without being intentional in the program elements and 
mechanisms they implement by using empirical evidence or theories of change in program 
design to affect those outcomes. It is clear that if our priority is to spend limited resources to 
provide supervision and activities for youth after school, we should also be investing in studying 
and implementing programs and program elements that are effective and grounded in empirical 
evidence and theory. Improving the design of the programs as well as the evaluations of ASPs to 
examine specific elements and contexts that may affect outcomes could provide valuable 
information to realize the potential of ASPs. 
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Appendix B. Tables and Figures 

Table 1 

Summary of Included Studies 


1 st Author 
(year) 

Program Name 

At-Risk 

Identifier 

Program 

Type 

Program 

Focus 

Weekly 

Contact 3 

N 

Grade 

Level b 

Study 

Design 

Outcomes 

Inclusion in 
Past Reviews 

Arcaira 

(2010) 

Citizen Schools 

High minority 

National 

Academic 

Not 

specified 

870 

2 

QED 

EB: Suspensions; AT: 
Attendance 

None 

Biggart 

(2013) 

Doodle Den 

Struggling 
readers, low- 
income 

Local 

Academic 

4 

621 

1 

RCT 

EB: Teacher reported 
ADHD Scale 

None 

Blumer 

(2010) 

Pathways 

History of 

high-risk 

behavior 

Local 

Non 

academic 

4 

37 

4 

QED 

EB: CAFAS 

None 

Foley 

(2001) 

Virtual Y 

High % low 
income and 
minority 

Local 

Mixed 

Not 

specified 

5915 

1 

QED 

AT: Grade 4 attendance 

Durlak et al. 

Frazier 

(2013) 

Project NAFASI 

High % 
minority and 
low income 

Local 

Non 

academic 

5 

127 

4 

QED 

EB: Aggression 

None 

Gottfredson 

(2010) 

All Stars 

High % 
minority and 
low income 

Local 

Non 

Academic 

4 

447 

2 

RCT 

EB: Last month drug 
use, disruptive 
behavior, aggression, 
delinquency 

None 

Gottfredson 

(2004) 

Maryland After 
School 
Community 
Grant 

High crime 
areas 

Local 

Non 

Academic 

4 

349 

4 

QED 

EB: Delinquency, 
rebellious, drug use 

Durlak et al. 
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Hirsch 

(2011) 

After School 
Matters 

High % 
minority and 
low income 

Local 

Mixed 

4 

535 

3 

RCT 

EB: Alcohol, drug use, 
risky intercourse, steal 
< $50, Steal > $50, 
suspension, sell drugs, 
carry weapon, fights 
gang activity; AT: 
attendance 

None 

James- 
Burdumy 
(2005, 2007, 
2008) 

21 st CCLC 

High % 
Minority 

National 

Not 

specified 

5 

2288 

1 

RCT 

EB: Suspensions; AT: 
Absences 

Apsler 
Durlak et al. 
Roth et al. 

James- 
Burdumy 
(2005, 2008) 

21 st CCLC 

High % 
Minority 

National 

Not 

specified 

4 

3831 

2 

QED 

EB: Student-reported 
disciplinary referrals, 
teacher reported 
discipline; AT: 
Absences 

Apsler 
Durlak et al. 
Roth et al. 

LaFrance 

(2001) 

Safe Haven 

High % 
minority and 
history of 
arrest 

Local 

Mixed 

5 

241 

4 

QED 

AT : Attendance 

None 

Langberg 

(2007) 

Challenging 

Horizons 

“Below basic” 
on 

standardized 

tests 

Local 

Academic 

4 

48 

2 

RCT 

EB: Parent CGI; Parent 
IRS; Teacher IRS 

None 

Le 

(2011) 

Roosevelt 
Village Center 

High % 
minority and 
immigrant 

Local 

Mixed 

5 

338 

2 

QED 

EB: Truancy; 
delinquency; arrest 

None 

Molina 

(2008) 

Challenging 

Horizons 

ADHD 

diagnosis 

Local 

Non 

Academic 

3 

20 

2 

RCT 

EB: Parent- 

externalizing; 

Adolescent- 

delinquency, 

maladjustment 

None 

Nguyen 

(2007) 

21 st CCLC 

Low income 
and academic 
achievement 

National 

Mixed 

4 

28169 

4 

QED 

EB: Disciplinary 
referrals, suspensions; 
AT: Absences 

None 
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Oyserman 

(2002) 

School to Jobs 

High % 
minority and 
low-income 

Local 

Non 

Academic 

2 

206 

2 

QED 

EB: Self-report 
referrals; AT: 
Attendance 

Durlak et al. 

Paschtel- 

Temple 

(2013) 

21 st CCLC 

Low academic 
achievement 

National 

Mixed 

3 

66 

2 

QED 

EB: Disciplinary 
referrals; AT: Absences 

None 

Prenovost 

(2001) 

After School 
Learning and 
Safe 

Neighborhoods 

Partnership 

High % 
minority, LEP 2 

Local 

Mixed 

5 

1358 

2 

QED 

AT: Absences 

Durlak et al. 
Lauer et al. 
Roth et al. 

Schinke 

(2000) 

Boys and Girls 
Club 

Low-income 

National 

Mixed 

5 

188 

4 

QED 

EB: Behavioral 
incidences; AT: 
attendance 

None 

Sibley 

(2004) 

21 st CCLC 

High % low- 
income and 
minority 

National 

Not 

specified 

Not 

specified 

78 

1 

QED 

EB: Discipline 
referrals; AT: 
Attendance 

None 

Smeallie 

(1997) 

Tutorial Club 

Academic 

failure 

Local 

Mixed 

3 

62 

2 

RCT 

AT : Attendance 

Lauer et al. 

Tebes 

(2007) 

Positive Youth 

Development 

Collaborative 

High % 
minority 

Local 

Non 

Academic 

Not 

specified 

304 

4 

QED 

EB: Alcohol, 
marijuana, other drug 
use 

Durlak et al. 

Weisman 

(2003) 

Maryland After 
School 
Community 
Grant 

High % 
minority 

Local 

Mixed 

4 

1068 

4 

QED 

EB: Rebellious 
behavior; delinquency; 
drug use; AT: days 
absent 

Durlak et al. 
Zief et al. 

Welsh 

(2002) 

The After School 
Corporation 

High % 
minority and 
low-income 

Local 

Not 

specified 

5 

68214 

4 

QED 

AT : Attendance 

Lauer et al. 
Scott-Little et al. 
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Table 2 


Study and Sample Characteristics 


Characteristic 

N (%) 

Characteristic 

N (%) 

Mean age 1 

11.7 

Predominant Race 


Free or reduced lunch 2 

78.4% 

Caucasian 

4 (16.7%) 

Percent male 3 

52.5% 

African American 

11 (45.8%) 

Grade level 


Hispanic 

1 (4.2%) 

Elementary 

4 (16.7%) 

Asian 

1 (4.2%) 

Middle school 

10(41.7%) 

Not reported 

7 (29.2%) 

High school 

1 (4.2%) 

Publication Year 


Mixed 

9 (37.5%) 

1990-1999 

1 (4.2%) 

Research Design Type 


2000-2009 

15 (62.5%) 

RCT 

7 (29.2%) 

2010-2014 

8 (33.3%) 

QED 

17 (70.8%) 

Publication Type 


Control Group Condition 


Journal 

11 (45.8%) 

Nothing or Waitlist 

17 (70.8) 

Dissertation or Thesis 

5 (20.8%) 

Treatment as Usual 

3 (12.5%) 

Government Report 

3 (12.5%) 

Specific Treatment 

4 (16.7%) 

Other Report 

5 (20.8%) 

Sample Size 


Country 


1-150 

7 (29.2%) 

United States 

23 (95.8%) 

151-300 

3 (12.5%) 

Ireland 

1 (4.3%) 

301-600 

5 (20.8%) 

Australia 

0 (0%) 

601 and greater 

7 (29.2%) 

Canada 

0 (0%) 



United Kingdom 

0 (0%) 

Note: 'Reported in 12 studies; 2 Reported in 10 studies; 

''Reported in 22 studies 



Table 3 

Intervention Characteristics 

Characteristic 

N (%) 

Characteristic 

N (%) 

Settings 


Number of Treatment Sessions 

School 

13 (54.2%) 

0-50 

3 (12.5%) 

Community-based organization 

5 (20.8%) 

51-100 

3 (12.5%) 

Mixed 

3 (12.5%) 

101-150 

7 (29.2%) 

Unsure 

3 (12.5%) 

15 land greater 

4 (16.7%) 

Program Focus 


Unsure 

7 (29.2%) 

Academic 

3 (12.5%) 

Length of sessions 


Non-academic 

5 (20.8%) 

1-1.59 hours 

4 (16.7%) 

Mixed 

12 (50.0%) 

2-2.59 hours 

4 (16.7%) 

Unsure 

4 (16.7%) 

3-3.59 hours 

10 (41.7%) 

Manual Used for Intervention 


4 hours and greater 

3 (12.5%) 

No 


10(41.7%) 

Unsure 3 

(12.5%) 




Yes, for entire program 

7 (29.2%) 

Weekly Contact Frequency 


Yes, for partial treatment 

6 (25.0%) 

Once 

1 (4.2%) 

Unsure 

1 (4.2%) 

Twice 

2 (8.3%) 

Program Coverage 


Three to Four 

9 (37.5%) 

National 

7 (29.2%) 

Five 

8 (33.3%) 

Local 

17 (70.8%) 

Unsure 

4 (16.7%) 
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Table 4: 


Moderator Analysis: Attendance Outcome 


Moderator 

Level (Effect Sizes) 

Effect Size (S.E.) 

95% C.I. 

Age 


Elementary (3) 

.06 (.04) 

-.27, .38 


Middle (7) 

-.02 (.06) 

-.20, .17 


Mixed (5) 

.07 (.03) 

-.06, .21 

Contact 


Weekly (2) 

-.22 (.26) 

-3.49, 3.05 


3-4x/ Week (4) 

.07 (.04) 

o 

<N 

MO 

O 


Daily (7) 

-.01 (.05) 

-.15, .15 

Control Type 


Wait list (13) 

.01 (.03) 

-.05, .08 


Treatment as usual (2) 

.06 (.11) 

-.15, .27 

Design 


Random (3) 

.04 (.04) 

-.30, .37 


Non-random (13) 

.04 (.03) 

-.03, .11 

Program Type 


Non-national (9) 

-.01 (.05) 

-.14, .12 


National (7) 

.06 (.03) 

-.02, .14 


Notes : None of the moderator analyses revealed significant differences (p < .05); 
“program focus” moderator eliminated from the analysis due to missingness. 
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Table 5: 


Moderator Analysis: Externalizing Behavior Outcome 


Moderator 

Level (Effect Sizes) 

Effect Size (S.E.) 

95% C.I. 

Age 

Elementary (3) 

.07 (.12) 

-.47, .62 


Middle (17) 

.14 (.06) 

-.01, .30 


Mixed (19) 

.15 (.26) 

-.52, .83 

Contact 

Weekly (4) 

.25 (.11) 

-1.17, 1.67 


3-4x/ Week (26) 

.02 (.06) 

-.13, .17 


Daily (13) 

.21 (.26) 

-.54, .95 

Control Type 

Wait list (27) 

.07 (.04) 

-.04, 36 


Treatment as usual (13) 

.81 (.67) 

-2.92, 1.77 


Straw-man (8) 

-.19(37) 

-2.16, 1.79 

Design 

Random (22) 

.07 (.09) 

-.23, .36 


Non-random (27) 

.14(31) 

-.10, .38 

Program Type 

Non-national (40) 

.04 (.07) 

-.11, .19 


National (9) 

.19(35) 

-.19, .56 

Focus 

Academic (5) 

.20 (.07) 

-.40, .75 


Non-academic (11) 

-.04 (.26) 

-1.04, .97 


Mixed (32) 

31(32) 

-.16, .38 


Notes'. None of the moderator analyses revealed significant differences (p < .05). 
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Figure 1. Study Selection Flow Chart 
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Figure 2. Forest Plot: Attendance Outcomes 
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Figure 2: Forest Plot: Externalizing Behavior Outcomes 
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